\ifnum\solutions=1 {
  \clearpage
} \fi
\item \subquestionpoints{2} \textbf{Relation to Newton's Method}


After going through all these steps to calculate the natural gradient, you might wonder if this is something used in practice. We will now see that the familiar Newton's method that we studied earlier, when applied to Generalized Linear Models, is equivalent to natural gradient on Generalized Linear Models. While the two methods (Netwon's and natural gradient) agree on GLMs, in general they need not be equivalent.


Show that the direction of update of Newton's method, and the direction of natural gradient, are exactly the same for Generalized Linear Models. You may want to recall and cite the results you derived in problem set 1 question 4 (Convexity of GLMs). For the natural gradient, it is sufficient to use $\tilde{d}$, the unscaled natural gradient.

\ifnum\solutions=1 {
  \input{03-natural_grad/06-update_rule_sol}
} \fi
